<!DOCTYPE html>
<html lang="en" dir="auto">

<head><script src="/livereload.js?mindelay=10&amp;v=2&amp;port=1313&amp;path=livereload" data-no-instant defer></script><meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="robots" content="noindex, nofollow">
<title>My First Post | 难赋小站</title>
<meta name="keywords" content="">
<meta name="description" content="1.LoRA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS（ICLR2022） 受两篇论文（Li et al. (2018a); Aghajanyan et al. (2020)）启发，认为We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach.
LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure 1.">
<meta name="author" content="">
<link rel="canonical" href="http://localhost:1313/posts/my-first-post/">
<link crossorigin="anonymous" href="/assets/css/stylesheet.4599eadb9eb2ad3d0a8d6827b41a8fda8f2f4af226b63466c09c5fddbc8706b7.css" integrity="sha256-RZnq256yrT0KjWgntBqP2o8vSvImtjRmwJxf3byHBrc=" rel="preload stylesheet" as="style">
<link rel="icon" href="http://localhost:1313/favicon.ico">
<link rel="icon" type="image/png" sizes="16x16" href="http://localhost:1313/favicon-16x16.png">
<link rel="icon" type="image/png" sizes="32x32" href="http://localhost:1313/favicon-32x32.png">
<link rel="apple-touch-icon" href="http://localhost:1313/apple-touch-icon.png">
<link rel="mask-icon" href="http://localhost:1313/safari-pinned-tab.svg">
<meta name="theme-color" content="#2e2e33">
<meta name="msapplication-TileColor" content="#2e2e33">
<link rel="alternate" hreflang="en" href="http://localhost:1313/posts/my-first-post/">
<noscript>
    <style>
        #theme-toggle,
        .top-link {
            display: none;
        }

    </style>
    <style>
        @media (prefers-color-scheme: dark) {
            :root {
                --theme: rgb(29, 30, 32);
                --entry: rgb(46, 46, 51);
                --primary: rgb(218, 218, 219);
                --secondary: rgb(155, 156, 157);
                --tertiary: rgb(65, 66, 68);
                --content: rgb(196, 196, 197);
                --code-block-bg: rgb(46, 46, 51);
                --code-bg: rgb(55, 56, 62);
                --border: rgb(51, 51, 51);
            }

            .list {
                background: var(--theme);
            }

            .list:not(.dark)::-webkit-scrollbar-track {
                background: 0 0;
            }

            .list:not(.dark)::-webkit-scrollbar-thumb {
                border-color: var(--theme);
            }
        }

    </style>
</noscript>
</head>

<body class="" id="top">
<script>
    if (localStorage.getItem("pref-theme") === "dark") {
        document.body.classList.add('dark');
    } else if (localStorage.getItem("pref-theme") === "light") {
        document.body.classList.remove('dark')
    } else if (window.matchMedia('(prefers-color-scheme: dark)').matches) {
        document.body.classList.add('dark');
    }

</script>

<header class="header">
    <nav class="nav">
        <div class="logo">
            <a href="http://localhost:1313/" accesskey="h" title="难赋小站 (Alt + H)">难赋小站</a>
            <div class="logo-switches">
                <button id="theme-toggle" accesskey="t" title="(Alt + T)">
                    <svg id="moon" xmlns="http://www.w3.org/2000/svg" width="24" height="18" viewBox="0 0 24 24"
                        fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"
                        stroke-linejoin="round">
                        <path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"></path>
                    </svg>
                    <svg id="sun" xmlns="http://www.w3.org/2000/svg" width="24" height="18" viewBox="0 0 24 24"
                        fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"
                        stroke-linejoin="round">
                        <circle cx="12" cy="12" r="5"></circle>
                        <line x1="12" y1="1" x2="12" y2="3"></line>
                        <line x1="12" y1="21" x2="12" y2="23"></line>
                        <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
                        <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
                        <line x1="1" y1="12" x2="3" y2="12"></line>
                        <line x1="21" y1="12" x2="23" y2="12"></line>
                        <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
                        <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
                    </svg>
                </button>
            </div>
        </div>
        <ul id="menu">
        </ul>
    </nav>
</header>
<main class="main">

<article class="post-single">
  <header class="post-header">
    
    <h1 class="post-title entry-hint-parent">
      My First Post
      <span class="entry-hint" title="Draft">
        <svg xmlns="http://www.w3.org/2000/svg" height="35" viewBox="0 -960 960 960" fill="currentColor">
          <path
            d="M160-410v-60h300v60H160Zm0-165v-60h470v60H160Zm0-165v-60h470v60H160Zm360 580v-123l221-220q9-9 20-13t22-4q12 0 23 4.5t20 13.5l37 37q9 9 13 20t4 22q0 11-4.5 22.5T862.09-380L643-160H520Zm300-263-37-37 37 37ZM580-220h38l121-122-18-19-19-18-122 121v38Zm141-141-19-18 37 37-18-19Z" />
        </svg>
      </span>
    </h1>
    <div class="post-meta"><span title='2024-05-24 09:43:35 +0800 CST'>May 24, 2024</span>

</div>
  </header> 
  <div class="post-content"><h1 id="1lora-low-rank-adaptation-of-large-language-modelsiclr2022">1.LoRA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS（ICLR2022）<a hidden class="anchor" aria-hidden="true" href="#1lora-low-rank-adaptation-of-large-language-modelsiclr2022">#</a></h1>
<p>受两篇论文（Li et al. (2018a); Aghajanyan et al. (2020)）启发，认为We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach.</p>
<p><img loading="lazy" src="my-first-post.assets/image-20240517085909058.png" alt="image-20240517085909058"  />
</p>
<p>LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure 1.</p>
<p>$h = W_0x + ∆W x = W_0x + BAx$</p>
<p><img loading="lazy" src="my-first-post.assets/image-20240517094732200.png" alt="image-20240517094732200"  />
</p>
<p>在做实验时只对注意力权重应用了LoRA。</p>
<p>做了实验1）探究对哪些层应用LoRA效果最好；2）r取几比较合适；3）$∆W$和$W$的相似性。</p>
<h2 id="11拓展">1.1.拓展<a hidden class="anchor" aria-hidden="true" href="#11拓展">#</a></h2>
<pre tabindex="0"><code class="language-mermaid" data-lang="mermaid">flowchart BT
LORA[&#34;1.LoRA&#34;]
MTIDOOL[&#34;1.1.1.Measuring&#34;]
IDETEOLMF[&#34;1.1.2.Intrinsic&#34;]
MTIDOOL --&gt; LORA
IDETEOLMF --&gt; LORA
MTIDOOL --&gt; IDETEOLMF
</code></pre><p>相关工作里的<strong>Low-Rank Structures in Deep Learning</strong>.可以帮助理解<strong>训练程度刻画idea</strong>。</p>
<p>GPT-3 175B的LoRA参数达到了一万倍的减少，这应该是由于，模型越大，需要的调整越少，因此需要引入的信息量越少，即需要的参数越少。</p>
<h3 id="111measuring-the-intrinsic-dimension-of-objective-landscapesicrl2018">1.1.1.Measuring the Intrinsic Dimension of Objective Landscapes（ICRL2018）<a hidden class="anchor" aria-hidden="true" href="#111measuring-the-intrinsic-dimension-of-objective-landscapesicrl2018">#</a></h3>
<p>模型参数是在高维空间（D）中进行更新的。本论文将模型参数更新限制在低维（d）中。不断增大d，直到关于某任务的解开始出现，将此时的维度称为该任务的固有维度（intrinsic dimension）。</p>
<h3 id="112intrinsic-dimensionality-explains-the-effectiveness-of-language-model-fine-tuningacl2021long">1.1.2.Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning（ACL2021.long）<a hidden class="anchor" aria-hidden="true" href="#112intrinsic-dimensionality-explains-the-effectiveness-of-language-model-fine-tuningacl2021long">#</a></h3>
<p>本篇是受1.1.1.的启发，前者研究了在从零开始训练的模型中的固有维度，本篇研究了预训练模型的固有维度。</p>


  </div>

  <footer class="post-footer">
    <ul class="post-tags">
    </ul>
  </footer>
</article>
    </main>
    
<footer class="footer">
        <span>&copy; 2024 <a href="http://localhost:1313/">难赋小站</a></span> · 

    <span>
        Powered by
        <a href="https://gohugo.io/" rel="noopener noreferrer" target="_blank">Hugo</a> &
        <a href="https://github.com/adityatelange/hugo-PaperMod/" rel="noopener" target="_blank">PaperMod</a>
    </span>
</footer>
<a href="#top" aria-label="go to top" title="Go to Top (Alt + G)" class="top-link" id="top-link" accesskey="g">
    <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 12 6" fill="currentColor">
        <path d="M12 6H0l6-6z" />
    </svg>
</a>

<script>
    let menu = document.getElementById('menu')
    if (menu) {
        menu.scrollLeft = localStorage.getItem("menu-scroll-position");
        menu.onscroll = function () {
            localStorage.setItem("menu-scroll-position", menu.scrollLeft);
        }
    }

    document.querySelectorAll('a[href^="#"]').forEach(anchor => {
        anchor.addEventListener("click", function (e) {
            e.preventDefault();
            var id = this.getAttribute("href").substr(1);
            if (!window.matchMedia('(prefers-reduced-motion: reduce)').matches) {
                document.querySelector(`[id='${decodeURIComponent(id)}']`).scrollIntoView({
                    behavior: "smooth"
                });
            } else {
                document.querySelector(`[id='${decodeURIComponent(id)}']`).scrollIntoView();
            }
            if (id === "top") {
                history.replaceState(null, null, " ");
            } else {
                history.pushState(null, null, `#${id}`);
            }
        });
    });

</script>
<script>
    var mybutton = document.getElementById("top-link");
    window.onscroll = function () {
        if (document.body.scrollTop > 800 || document.documentElement.scrollTop > 800) {
            mybutton.style.visibility = "visible";
            mybutton.style.opacity = "1";
        } else {
            mybutton.style.visibility = "hidden";
            mybutton.style.opacity = "0";
        }
    };

</script>
<script>
    document.getElementById("theme-toggle").addEventListener("click", () => {
        if (document.body.className.includes("dark")) {
            document.body.classList.remove('dark');
            localStorage.setItem("pref-theme", 'light');
        } else {
            document.body.classList.add('dark');
            localStorage.setItem("pref-theme", 'dark');
        }
    })

</script>
</body>

</html>
